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DESCRIPTION 

2-D MESH GEOMETRY AND MOTION VECTOR COMPRESSION 

Rftlatftri Applications 
U.S. Patent AppUcation Serial No. 08/447,418, filed May 23, 1995 for: "Video 
Compression System"; and 

U.S. Patent Application Serial No. 60/010,076, filed January 16, 1996 for 
"Video Coding and Decoding System". 

Figld of thg Invgnnon 
This invention is in the field of coding and manipulation of images, and 
specifically, the coding of sequences of 2D meshes, generally corresponding to a 
sequence of images. 

Background of the Invention 

A 2D triangular mesh refers to a tessellation of a 2D visual object plane 
mto triangular patches. The vertices of the triangular patches are called "node points." 
The straight-line segments joining the node points are called "edges.'* 

A dynamic 2D mesh consists of a temporal sequence of 2D triangular 
meshes, where each mesh has the same topology (i.e., structure), but node positions may 
differ from one mesh to the next Thus, a dynamic 2D mesh may be defined by the 
geometry of the initial 2D mesh and motion vectors at the node points for subsequent 
meshes, where each motion vector points from a node point of the previous mesh in the 
sequence lo a node point of the current mesh. The dynamic 2D mesh may be used to 
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create 2D animations by mapping texture from a still image onto successive 2D meshes 
via well-known texture mapping methods. For example, the dynamic mesh may be used 
to render a waving flag from a still image of a flag. The local deformations of the 
texture in time are captured by the motion of mesh nodes from one mesh to the next 

5 Hence, different animations of the same texture may be achieved by different sets of 
node motion vectors. 

Texture mapping utilizes the strucmre of the mesh, i.e., the way the nodes 
of the mesh are connected with each other, namely the configuration of the edges of the 
mesh. A mesh may have a specified implicit structure, such as uniform structure or 

10 Delaunay structure, as described in S. M. Omohundro, *The Delaunay triangulation and 
function learning," International Computer Science Institute Technical Report TR-90- 
001, University of California Berkeley, January 1990. 

Efficient coding of an animation sequence may be achieved by separately 
coding the still image texture, and the associated 2D mesh, i.e., the geometry and node 

15 vectors. The associated 2D mesh is represented by the geometry of the first mesh and 
motion vectors of the nodes of this first and subsequent meshes. The 2D mesh is 
encoded by coding the geometry of the first mesh and motion vectors of the nodes of 
this first and subsequent meshes. 

The mesh geometry compression technique described here is limited to 2D 

20 triangular meshes with implicit topology, specifically meshes with uniform and Delaunay 
topology. In these cases, the mesh topology is defined implicitly, given the locations of 
the mesh nodes (also called vertices) and some additional information to be specified in 
detail later. Algorithms to implement Delaunay triangulations are available in literature 
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and are not described here. It should be noted that Delaunay trianguiations are uniquely 
defined except if the nodes to be triangulated contain certain degeneracies in their 
locations. Here, it is assumed that both the mesh encoder and decoder use an agreed 
upon technique to handle such degeneracies. Such teclmiques are well known to those of 
5 skill in the art The mesh geometry compression technique described here allows a high 
compression ratio for these constrained classes of meshes. 

Representing mesh motion efficiently is important for describing mesh- 
based animations. Here, we describe a technique for compression of mesh motion in the 
2D case, although it should be noted that the principle may be extended to the case of 3D 

10 meshes with 3D motion straightforwardly. Furthermore, it should be noted that the mesh 
motion compression technique described here is directly applicable to meshes with 
general topology, although the examples provided herein describe meshes with 
constrained topology. Finally, it should be noted that the principles of the invention with 
respect to motion coding may also be applied to the coding of surface appearance 

15 attributes straightforwardly. 

The coding methods described here may for instance be employed in the 
context of MPEG-4. MPEG-4 is an object-based multimedia compression standard being 
developed by the Motion Picture Experts Group, which allows for encoding of different 
audio-visual objects (AVO) in the scene separately, as an extension of the previous 

20 MPEG- 1/2 standards. These AVO are decoded and then composited at the user terminal 
according to a transmitted scene description script and/or user interaction to form display 
frames. The visual objects may have natural or synthetic content, including audio, video, 
3D graphics models, scrolling text and graphics overlay, and so on. 
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Summary of the Invention 
This invention comprises methods for encoding the mesh data into a 
compressed format and methods for decoding the compressed format The use of a 
compressed format facilitates efficient storage and commmiication of the mesh data. The 
coding methods described are lossless, Le. the compressed format represents the same 
information contained in the original mesh data. At the same time, use of the 
compressed fonnat reduces the amount of storage space or communication bandwidth 
required. 

The inventions includes system method of encoding and decoding a 
dynamic mesh, including encoding and decoding a mesh geometry of a set of node 
points; and encoding and decoding a mesh node motion vector for each node point 

An object of the invention is to provide a system and method for encoding 
and decoding the mesh and displacement of the node points from one frame time instant 
to the next 

Brief Descri ptinn nf the Drawings 
Fig. 1 depicts a mesh-based motion modeling using a triangular mesh. 
Fig. 2 depicts node motion and deformation of an image object when a 
mesh is imposed thereover. 

Fig. 3 depicts a typical encoder/decoder system. 

Fig. 4 depicts an animation application being processes by the system of 

Fig. 3. 

Fig. 5 depicts a generalized uniform 2D mesh. 
Fig. 6 depicts four types of uniform meshes. 
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Fig. 7 depicts a node point ordering in a 2D triangular mesh of the 
Delaunay type. 

Fig. 8 depicts node point ordering in a 2D triangular mesh of the uniform 

type. 

5 Fig. 9 depicts a halfway-point in a breadth-first traversal of a 2D 

triangular mesh. 

Fig. 10 depicts a breadth-first traversal of triangles in a 2D mesh. 
Fig. 1 1 depicts a set of decoded node points and mesh boundary edge 

segments. 

10 Fig, 12 depicts a decoded triangular mesh obtained by constrained 

Delaunay triangulation. 

Detailed Da^CTiptinn of the. Preferred Embodiment 

Mesh-Based Motion Modeling for Video Processing 

Modeling and estimating the motion of objects in a sequence of image 
15 frames is common in video processing and has a variety of appUcations. A common 
approach is block-based motion modeling, where motion parameters are estimated for 
each square block of pixels independently. SpecificaUy, the translational block motion 
model has proven satisfactory for video compression; and has been adopted in 
international standards such as MPEG-1, -2 and -4, Recently, 2D mesh-based motion 
20 modeling has been proposed as a promising alternative in video processing to block- 
based motion modeling. 

A 2D mesh is a tessellation (or partition) of a 2D planar region into 
polygonal patches. In the following, the vertices of the polygonal patches are referred to 
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as the node points of the mesh. The patches are usually triangles or quadrangles, leading 
to triangular or quadrilateral meshes, respectively. See Fig. la. In 2D mesh-based 
motion modeling, a 2D mesh is associated with an image or a set of images, such that the 
polygonal mesh patches correspond to patches of texture in the images, where the texture 

5 consists of the image pixels inside a patch. Polygonal patches in a reference frame are 
deformed by the movements of the node points into polygonal patches in another frame, 
and the texture inside each patch in the reference frame is warped onto the other frame 
using a parametric mapping as a function of the node point motion vectors. See Figs, lb 
and Ic. For triangular meshes, affine mapping is- used, which may model translation, 

10 rotation, scaling and shear. Note that, in mesh-based motion modeling, the patches do 
not overlap in the reference frame or in the current frame. As used herein, a statement 
that two triangles are adjacent means that they share a common edge. 

Assuming that proper constraints are imposed in the parameter estimation, 
an affine transform can guarantee the continuity of the mapping across the boundaries of 

15 adjacent triangles. This unplies that the original 2D motion field may be compactly 

represented by the motion of the node points, from which a continuous, piece-wise affine 
motion field can be reconstructed. 

3D polygon meshes have long been used for efficient 3D object geometry 
modeling and rendering in computer graphics. Equations similar to parametric mappings 

20 used in mesh-based motion modeling have also been used in 3D graphics to perfoim 
texture mapping, a popular procedure to render natural images on polygon meshes 
describing graphic objects for photo-realistic synthesized images. Texrare mapping in 
3D graphics is realized by assigning a texture coordinate (a pixel position on a 2D image) 
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to every 3D node point on the polygonal mesh. Thus, each polygonal surface element on 
the 3D mesh is associated with a patch of the 2D image, which is then rendered on the 
polygon mesh subject to proper warping transformation. An animation may be created 
by rendering the same image onto a deforming mesh repeatedly. A similar process may 

5 be performed with 2D meshes to render an animated image sequence from an image and 
corresponding deforming meshes. 
Motion Estimation for Mesh-Based Video Processing 

Determining the motion of 2D meshes from a given image sequence is 
performed by estimating the motion of mesh node points over time. In the case of 2D 

10 mesh-based motion modeling, motion estimation refers to searching in a given reference 
image for the best locations of the node points, such that die ttiangular hnage patches in 
the reference frame optimally match those in the current image. The mesh in the initial 
reference image may have a regular structure, in which case it is called a uniform mesh, 
or it may be adapted to the image, in which case it is called a content-based mesh. A 

15 description of an algorithm for content-based (adaptive) mesh design may be found in the 
literature. 

Various techniques have been proposed for node motion vector search 
from one frame to the next The simplest method is to form blocks that are centered 
around the node points and then use a gradient-based technique or block-matching to find 
20 motion vectors at the location of the nodes. Hexagonal matching and closed-form 
matching techniques find the optimal motion vector at each node under the parametric 
warping of all patches surrounding the node while enforcing mesh connectivity 
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constraints. Another method is iterative gradient-based optimization of node point 
locations, taking into acconnt image features and mesh deformation criteria. 

Referring now to Fig. 2, an image object 10 is depicted. The motion of 
nodes, such as nodes 12, 14, from an image object 10 is described as the image object is 

5 transformed into an animated image object 16, having nodes 12a, 14a. The motion of 
nodes describe local motion and deformation of image object 10 over which a mesh 18. is 
imposed. The transformation that takes the mesh of object 16 to image object 10 results 
in local motion and deformation as well as a scaling of the entire image object 

As previously noted an object of this invention is to provide a method for 

10 encoding and decoding the mesh and displacement of the node points from one frame 
time instant to the next. The texture to be animated may be encoded using a still image 
coding method, such as those used in MPEG-4 (Moving Picture Experts Group Standard 
4). The texture to be animated may be a still image of a natural object or it may be a 
synthetic (computer generated) image. 

15 The application at the decoder decodes the texmre and the mesh and 

renders the animation. The current version of MPEG-4 Systems Working Draft (WD) 
specify a framework for applications that support animation, MPEG-4 Systems Working 
Draft V 5.0 of ISO/IEC 14496, Document No. N1825, July 1997, hereinafter MPEG-4 
WD V 5.0). 

20 2D Mesh Encoder 

Initially, the encoding of the mesh geometry and then encoding of mesh 
motion vectors will be described. Assume that the mesh structure (i.e., topology) is 
known and is either a uniform mesh or a Delaunay mesh. 

8 
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Referring to Figs. 3 and 4, a typical system that may use the method of the 
invention is depicted, and includes an encoder subsystem,. 20, shown in Fig. 3. 
Subsystem 20 includes a texture encoder 22, which receives still image texture data, 24, 
and generates a texture coded bit stream, 26. A mesh encoder 28 receives a 2D mesh 

5 sequence 30. Encoder 28 includes a geometry encoder 32 and a motion vector encoder 
34. Encoder 28 generates a mesh coded bit stream 36. 

The decoding subsystem 20 is shown in Fig. 4, and includes a texture 
decoder 38 and a mesh decoder 40, Mesh decoder 40 includes a geometry decoder 42 
and a motion vector decoder 44. Output from texture decoder 38 and mesh decoder 40 

10 results in texture mapping data 46, which is used in an animation application 48. 

resulting in an animated image 50. It will be appreciated by those of skill in the art that 
while the preferred embodiment of the system of the invention is described as 
manipulating a still image, other images, such as video images, may also be manipulated, 
Mf><;h| rTfiomfttrv Encoding 

15 Because the initial 2D triangular mesh is either a uniform mesh or a 

Delaunay mesh, the mesh triangular topology (links between node points) is not encoded; 
only the 2D node point coordinates p^ = (x„, yj are encoded. In the bit stream, a special 
flag may specify whether the initial mesh is uniform or Delaunay. See Table 8, below. 

In case of a uniform mesh, five parameters, nr_of_mesh_nodes_hori, 

20 nr_or_mesh„nodes„verti, mesh_rect_size_hori, mesh_rech_size_verti, and 
triangle_split_code, (further defined and described ia Table 5, below) are used to 
specify the complete topology and node point locations. An example of a 2D uniform 
mesh is shown in Fig. 5, generally at 60, wherein the number of nodes 62, shown as dark 
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circles for emphasis, in the horizontal and vertical dimensions are equal to 5 and 4, 
respectively. The splitting of rectangles into triangles may be done in four different 
schemes, as shown in Fig. 6. One of these types is shown in Fig. 5, which corresponds to 
triangle_split_code = '00'. Other codes are further defined in Table 9. The first two 

5 parameters, , nr_of.mesh_nodes_hori, iir_or_mesh.nodes_verti, specify the number 
of nodes in the horizontal and vertical direction, respectively, of the uniform mesh. The 
next two parameters, mesh_rect„size_hori, mesh_rect_size„verti, specify the 
horizontal, and vertical size of each rectangle (containing two triangles) in half pixel 
units, respectively. This specifies the layout and dimensions of the mesh. The last 

10 parameter, triangIe_spIit_code, specifies how each rectangle is split to form two 
triangles. 
Delamay mesh 

The node point coordinates are encoded, by first encoding the boundary 
node points and then the interior node points of the mesh. To encode the interior node 

15 positions, the nodes are traversed one by one using a nearest neighbor strategy and each 
node position is encoded differentially using the position of the previously encoded node 
as a predictor. A linear ordering of the node points is computed such that each node is 
visited once. When a node is visited, its position is differentially encoded with respect to 
the previously encoded node. That is, the difference between the position of the present 

20 node and the reconstructed value of the previous node is encoded using variable length 
coding (VLC). The ordering is such that the boundary nodes are visited first; then the 
interior nodes. By sending the total number of node points and the number of boundary 
node points, the decoder knows how many node points will follow, and how many of 
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those are boundary nodes; thus it is able to reconstruct the polygonal boundary and the 
locations of all nodes. 

This procedure is illustrated in Fig. 7, which is an example of a small 
triangular mesh, 70. Fig. 7 depicts a traversal of node points of a 2D triangular mesh and 

5 ordering of the node points to be coded. First, the boundary nodes, Po—Pj, are visited 
according to connectivity, i.e., the next node is alv^ays the next connected node on the 
boundary in the counterclockwise direction. Then, the interior nodes, p^o ...p^^, are visited 
according to proximity, i.e., the next node is always the nearest node that is not already 
encoded. The total number of nodes and the number of boundary nodes is encoded. The 

10 top-left node po is encoded without prediction, where the top left node Po = (Xq, yo) is 
defined as the node n with minimum H-y^, assuming the origin of the local coordinate 
system is at the top left. If there is more than one point with the same value of \ + y^, 
then choose the node point with minimum y. Then, the next counterclockwise boundary 
node Pi is found and the difference between Po and p^ is encoded; then all other 

15 boundary nodes are encoded in a similar fashion. Then, the not previously encoded 

interior node that is nearest to the last boundary node is found and the difference between 
these is encoded. That node is defined as the not already encoded node n with minimum 

- x^l H-|y^ - yi^l , where (x^^ ,y,^ ) represent the coordinates of the previously 
encoded node. Then, the not previously encoded node nearest to the last encoded node is 

20 found and the difference is encoded, and so on. Every node point has an x- and y- 

coordinate, p^ = (x^, y^) , each of which is subtracted from the corresponding coordinate 
of the previously encoded node pouit The two resulting difference values are encoded 



11 



wo 98/27515 



PCT/JP97/04607 



using variable length coding (VLC). In a particular embodiment of the current invention, 
variable length codes specified by MPEG-4 for coding sprite trajectories are used. In 
principle, specific variable length codes may be designed. 
Fncoding of Node M otion Vectors 
5 Each node point p„ of a 2D mesh numbered kin the sequence of meshes 

has a 2D motion vector v^ , defined from mesh * to ^+1. The locations of node points p^' 

of the mesh numbered Jk+l are encoded by encoding all motion vectors v^ , n = 0 N - 

1. Note that the triangular topology of the mesh remains the same throughout the 
sequence. 

10 Two alternative methods for node motion vector coding will be described, 

both of which incorporate predictive coding of the node motion vectors. Predictive 
coding of motion vectors entails prediction of each motion vector by one or more already 
encoded motion vector(s) of other node points in the mesh; subsequently the difference 
between the predicted vector and actual motion vector is encoded instead of the original. 

15 The first method, Method I, uses only one already encoded motion vector to predict the 
value of a particular motion vector. The second method. Method n, uses two already 
encoded motion vectors to predict the value of a particular motion vector. The methods 
also differ with respect to the choice of predicting motion vectors; in Method I the 
predicting motion vector is defined as the preceding motion vector in a simple node point 

20 ordering; in Method II the predicting motion vectors are defined by the use of a breadth- 
first traversal of the mesh. Note that the bit stream syntax for mesh motion coding, as 
defined in Table 6, is the same for Method I and n. 
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Method T 

Assume there is a urdque ordering of the available node points p^, where 
n denotes the ordering number. In the case where the mesh is a Delaunay-type mesh, as 
shown in Fig. 7. the ordering is simply defined as the order in which the node point 
5 locations are encoded during mesh geometry encoding; i-e., the node point for which the 
location is encoded first is Pq, the node point for which the location is encoded after that 
is Pi, etc. The ordering used during mesh geometry encoding is based on a traversal of 
the initial mesh, where boundary nodes are visited first, then interior nodes are visited, as 
previously described. 

10 In the case where the mesh is a uniform-type mesh, as shown in Fig. 8, the 

node ordering is defined on the basis of die mesh structure as follows. The first node 
point in the ordering, po , is the most top-left node point of the uniform mesh; the 
following node points in the ordering are obtained by traversing the node points of the 
unifomi mesh from left to right in the first (top) row of node points; going to the node in 

15 the second row of nodes immediately below the last node of the first row; traversing the 
second row from right to left; going down to the third row of nodes and traversing the 
third row from left to right, etc. This ordering is defined by a simple traversal of the 
uniform mesh. 

For each node point in the mesh, a one bit flag, the 
20 node_motion_vector_flag, is encoded to specify whether that node point has a non-zero 
motion vector. For each node point in the mesh with non-zero motion vector, a motion 

13 
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vector difference vector is encoded to specify the difference between the motion 
vector of diat node and the previously encoded motion vector. 

Thus, the encoding process is as follows: The motion vectors of all node 
points p^, /I = 0, AT - 1, are defined by: 

v.=^r-p«. (1) 
If the first node point in the ordering defined above has a zero motion vector, i.e. 
Vq = (0,0), no motion vector data is encoded for this node point. If the first node point 
has a non-zero motion vector, then a vector e^ is defined, equal to the motion vector of 
the first node point: 

eo=Vo. (2) 
The vector is encoded by VLC of its x- and y-component. For all other node points in 
the ordering as defined above, the encoding process is as follows: 

If node point p^, /z = 1, A^- 1, has a zero motion vector, Le. v^ = (0,0)» 
then no motion vector data is encoded for that node point. Otherwise, a motion vector 
difference vector ? is computed from the previously encoded motion vector and the 

current motion vector by: 

en=v,-Vi. (3) 
Then, the vector e^ is encoded by variable length encoding of its x- and y-component 
Thus, to encode the actual motion vectors, difference vectors e„ are encoded one by one 
in the order specified above. 
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Method n: M nHnn Vector Prediction 

Method n is based on a predictive coding technique where nvo motion 
vectors are used to compute a predictor. The predictive coding technique of Method n 
specifically employs the following technique of motion vector prediction. To encode the 

5 motion vector of a node point p^ that is part of a triangle t^ = (pj, p^, p^), where the two 
motion vectors vectors Vj and v„ of the nodes Pi and have already been encoded, one 
may use the values of Vi and v„ to predict v„ and encode the prediction error value. 
Starting from an initial triangle tj,, for which all three node motion vectors have been 
encoded, there must be at least one other, neighboring, triangle t^ diat has two nodes in 

10 common with tj^. Because the motion vectors of the two nodes that t^ and t^ have in 

common have already been encoded, one may use these two motion vectors to predict the 
motion vector of the third node in t^. The actual prediction vector w^ is computed by 
averaging of the two prediction motion vectors and the components of the prediction 
vector are rounded to half-pixel accuracy, as follows: 

15 w, -0.5x(Lv„+Vi+0,5j). (4) 

This prediction formula is used for all motion vectors, except for the first and second 
motion vectors encoded. When encoding the first motion vector, the prediction vector is 
set to the zero vector: 

w,=(0,0). (5) 
20 When encoding the second motion vector, the prediction vector is set to the first encoded 
motion vector 

= v,^ (6) 
15 
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For each node point n, a prediction error vector, e^, is computed by taking the difference 
between the prediction vector and the actual motion vector 

ea=v„-w^. (7) 
Each prediction error vector is encoded using variable length encoding. This procedure 

5 is repeated while traversing the triangles and nodes of the mesh, as explained below. 
Note that the prediction error vector is encoded only for node points with a non-zero 
motion vector. For all other node points, the motion vector is simply v^ = (0,0). The 
previously identified node_motion_vector_flag is used to specify whether a node has a 
non-zero motion vector or not. 

10 Method H; M^^sh Trav^rs^ 

A breadth-first traversal is used to visit all the triangles and nodes in the 
mesh numbered Jfc, and to encode the motion vectors defined from mesh to ^+1. The 
breadth-first traversal is a uniquely defined traversal of the mesh such that every triangle 
is visited exactly once and each node is visited at least once. Because this traversal is 

15 determined by the topology of the mesh, which is known at the encoder at all times and 
known at the decoder once the initial mesh has been decoded, the traversal of the mesh 
may be performed at both the encoder and decoder in exactly the same manner. 

The breadth-first traversal of the mesh triangles is defined as follows and 
is illustrated in Fig. 9, generally at 90. First, define an initial triangle 92 as follows: 

20 Define the top left mesh node 94 as the node n with minimum + y^ , assuming the 
origin of the local: coordinate system is at the top left. If there is more than one node 
with the same value of + y^ , then choose the node point among these with minimum y. 



16 



wo 98/27515 



PCT/JP97/04607 



The edge 96 between the top-left node of the mesh and the next clockwise node 98 on the 
boundary is an edge of the triangle that is designated as the initial triangle. Label the 
initial triangle with the number 0. 

Second, all other triangles are iteratively labeled with numbers 

5 1, 2, M - 1, where M is the number of triangles in the mesh. Among all labeled 

triangles that have adjacent triangles which are not yet labeled, find the triangle with the 
lowest number label. This triangle is referred to in the following as the current triangle 
100. In the first iteration, the initial triangle becomes the current triangle. Define the 
base edge of this triangle as the edge that connects this triangle to die abready labeled 

10 neighboring triangle with the lowest number. In the case of the initial triangle, the base 
edge is defined as the edge between the top-left node and the next clockwise node on the 
boundary. Define the right edge of the current triangle as the next counterclockwise 
edge with respect to the base edge; and define the left edge as the next clockwise edge 
with respect to the base edge. That is, for a triangle = (pi, p„, p„), where the vertices 

15 are in clockwise order, if (pipj is the base edge, then (pipj is the right edge and 

(PmPn) is the left edge. 

Third, determine if there is an unlabeled triangle adjacent to the current 
triangle, sharing the right edge. If fliere is such a triangle, it is labeled with the next 
available number. Determine if there is an unlabeled triangle adjacent to the current 
20 triangle, sharing the left edge. If there is such a triangle, it is labeled with the next 
available number. This process is continued iteratively until all triangles have been 
labeled with a unique number m. Referring now to Fig. 9, a breadth-first traversal of a 
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2D triangular example mesh is depicted. The traversal is halfway through the mesh - 
five triangles have been labeled (with numbers) and the motion vectors of six node 
points have been encoded (marked with a box symbol). The triangle which has been 
labeled '3' is the 'current triangle'; the base edge is *b'; the right and left edge are *r* and 
5 *1'. The triangles that will be labeled next are the triangles sharing the right, resp. left 
edge with the current triangle. After those triangles are labeled, the triangle which has 
been labeled *4' will be the next ^current triangle' and another motion vector will be 
encoded 

The ordering of the triangles^ according to their assigned label numbers 
10 implicitly defines the order in which the motion vector data of each node point is 

encoded. Initially, motion vector data for the top-left node of the mesh is encoded. No 
prediction is used for the motion vector of this node, hence this data speicifies the motion 
vector itself. Then, motion vector data for the second node, which is the next clockwise 
node on the boundary with respect to the top-left node, is encoded. This data contains 
15 the prediction error for the motion vector of this node, where the motion vector of the 
top-left node is used as a prediction. These first two nodes that form the base edge of the 
initial triangle are marked with the label 'done'. 

During each iteration of the breadth-first traversal, as described above, 
and starting fi*om the initial triangle, the motion vectors of the two nodes that are on the 
20 base edge of the current triangle during that iteration are used to form a prediction for the 
motion vector of the third node of that triangle. If that third node is not yet labeled 
'done', prediction error values are computed by subtracting the prediction from the actual 
motion vector, and the motion vector data is encoded by VLC. The third node is labeled 
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'done' . If the third note is already labeled *done' , it is simply ignored and no data is 
encoded. Note that due to the nature of the traversal process, the two nodes on the base 
edge of a triangle are guaranteed to be labeled *done' when that triangle becomes the 
'current triangle', signifying that their motion vectors have already been encoded and 
may be used as predictors. Motion vectors and triangle queue states for the encoding 
steps of the method are shown in Table 1. 



encode 




triangle queue state 






{t3} 






{ts, to} 


V7 




{to. ta} 


Vo 




{ts. ta} 


Vs 




{ta, t9. te} 


V4 




{tg, ts, ti} 


V9 




{te, ti} 






{ti, tr} 


V2 




{t7, t4} 


Vs 




{U} 






{} 



Table 1 



The breadth-first traversal of the triangles and the encoding of the node 
motion vector data may be performed simultaneously by maldng use of a first-in-first-out 
(FIFO) queue of triangles. It should be noted, however, that the following description 
serves as a preferred embodiment of the invention, and that implementation of the 
motion vector encoding process using a triangle queue is only one of several ways to 
implement the breadth-first traversal. 
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The FIFO queue is used to temporarily store triangles t^, where each 
triangle is specified by an ordered triple of node points tj, = (pi, Pxa> Pn)> such that the 
node points of the triangle are stored in clockwise order and such that the first and second 
node points always correspond to the base edge of a triangle. 

The top-left node of the mesh and its neighboring node on the boundary, 
in a clockwise direction, together define the initial triangle in the traversal. The motion 
vectors associated with these two nodes are encoded as described above. The (initially 
empty) FIFO queue is now initialized by appending the initial triangle at the end. 
Furtheimore, the initial triangle is labeled with number 0, and its two already-processed 
nodes are marked 'done\ Next, the FIFO queue is processed as foUows until it is empty. 

The ordered triple t,^ = (pp p^, p^) that is at the head of the queue is 
removed from the queue. The base edge, right edge and left edge of this triangle may be 
identified immediately from this triple. If the third node, p„, is marked 'done', no 
further action is taken; otherwise, a prediction vector is computed using the already 
encoded motion vectors associated with the first two nodes in the triple, p^ and p^, and 
the acmal motion vector at p^ . The prediction error values are encoded by VLC. The 
subject node point now marked *done'. Determine whether t,^ has an adjacent triangle 
sharing the right edge that has not yet been labeled. If there is such a triangle, it is 
labeled with a new number and appended to the end of the queue. Determine whether tj^ 
has an adjacent triangle sharing the left edge that has not yet been labeled. If there is 
such a triangle, it is labeled with a new number and appended to the end of the queue. 
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When all triangles have been labeled with a number, no more triangles 
will be appended to the queue. When the queue is empty, all triangles will have been 
processed and all node motion vectors will have been encoded. 

Because every triangle in the mesh has at least one adjacent neighbor, and 

5 triangles are labeled when visited, every triangle is visited exactly once and the traversal 
terminates when (and only when) all triangles have been visited. The breadth-first 
traveisal of triangles defines an ordering in which node points are visited. Node points 
may be visited more than once, but their motion vectors are encoded only at the first time 
a node is visited. Each node is labeled at the time of encoding, such that no attempt will 

10 be made to encode the corresponding motion vector again. The unique ordering of node 
points corresponds to the order of motion vector data put in the bit stream. 

The breadth-first traversal process is illustrated in Fig. 10 for a small 
triangular mesh, showing a traversal of triangles arbitrarily numbered to , and the 
corresponding ordering of node points arbitrarily numbered po, pg. The initial 

15 triangle is defined using the top-left node pj and the next clockwise node on the 

boundary p^. The motion vector V3 is encoded without using any prediction; the motion 
vector Vj is encoded using only V3 as a prediction. Nodes pg and Pi are marked 'done'. 
The triangle queue is initialized with tg and ig is labeled. The actual breadth-first 
traversal of the mesh now starts by removing the front element from the queue, in this 

20 case tg . The motion vector of its third node, v^ , is now encoded using V3 and Vi as 

predictors and putting the prediction error in the bit stream. Node p^ is marked 'done\ 
The next step is to append any triangles adjacent to the current triangle tg and not yet 
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labeled to the triangle queue. The triangle on the right of the current triangle (in this case 
tj) is appended first; then the triangle on the left (in this case to); both are labeled. 

The next iteration of the traversal commences by removing the front 
element from the queue, in this case . The motion vector of its third node, v, , is now 
encoded using V3 and % as predictors and retrieving the prediction error from the bit 
stream. Node p^ is marked *done' and the connected triangle tj is appended to the 
triangle queue and labeled. 

Such iterations continue until the triangle queue is empty and all node 
motion vectors have been encoded. The steps of the algorithm are illustrated in Table 1. 
2D Mesh Decoder 

In reference to the MPEG-4 syntax, described later herein, and again 
referring to Fig. 4 and Tables 3-6, after mesh_object,start_code has been decoded by 
mesh decoder 40, a sequence of mesh object planes is decoded, until a 
mesh_object_end_code is detected. The new.mesh^flag of the mesh object plane class 
determines whether the data that follows specifies the initial geometry of a new dynamic 
mesh, in which case the data is sent to geometry decoder 42, or whether it specifies the 
motion of nodes from the previous mesh to the current mesh, in a sequence of meshes, m 
which case the data is sent to motion vector decoder 44. 

In the following, the decoding of mesh geometry is described, followed by 
a description of the decoding of mesh motion. This description is general in nature and 
may be implemented using any syntax other than the MPEG-4 syntax specified herein in 
the preferred embodiment 
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Mesh Geometry Decoding 

Because the initial 2D triangular mesh is either a uniform mesh or a 
Delaunay mesh, the mesh triangular topology (links between node points) is not encoded; 
only the 2D node point coordinates p^ = (x^, y^) are encoded. The mesh_type_code 
5 (Table 8) specifies whether the initial mesh is uniform or Delaunay. 
Uniform mesh 

As previously stated, five parameters specify the geometry of a uniform 
mesh (Table 5). The first two decoded parameters specify the number of nodes in the 
horizontal and vertical, respectively, direction of the uniform mesh. The next two 
10 decoded parameters specify horizontal and vertical size of each rectangle (containing two 
triangles) in units accurate to half pixel units. The last parameter specifies how each 
rectangle is split into two triangles. 
Delmay mesh 

First, the total number of node points in the mesh A*^ is decoded; then, the 
15 number of node points that are on the boundary of the mesh //^ is decoded. Note that N 
is the sum of the number of nodes in the interior of the mesh, A^^ and the number of nodes 
on the boundary, iV^, 

N = Ni+Ni,. (8) 
Then, the locations of boundary and interior node points are decoded. The origin of the 
20 local coordinate system is presumed to be at the top left of the image objecL 

The X-, and y-coordinates of the first node point are decoded, which is 
specified in half-pixel units by a fixed length code (FLC). All the other node point 
locations are computed by adding differential values to previously decoded node 
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locations. In particular, a delta_x, and deltajr value is added to the x- and y- 
coordinate, respectively, of the previously decoded node location. The delta_x and 
delta_y value are each decoded by first decoding a variable length code (VLC) 
specifying their respective lengths, delta_x Jen„vlc and delta _y Jen_vlc, and then 
5 decoding their values. Thus, the coordinates of the initial node point Po = (x^, yo) is 
decoded as is. The coordinates of all other node points, p^ = (x^, y^ ) are determined by 
adding a decoded value to the previously decoded node poiQt coordinates: 
X -X ^ +dx and y = y , + 4y» • The differential values are decoded using variable 

length codes used at the encoder. 

10 The ordering in the sequence of decoded locations is such that the first A^,^ 

locations correspond to boundary nodes (in counterclockwise direction). Thus, after 
receiving the first locations, the decoder is able to reconstruct the boundary of the 
mesh by connecting each pair of successive boundary nodes, as well as the first and the 
last, by straight-line edge segments. The next N - values in the sequence of decoded 

15 locations correspond to interior node points. Thus, after receiving N nodes, the locations 
of both the boundary and interior nodes may be reconstructed, in addition to the 
polygonal shape of the boundary. This is illustrated in Fig. 11, which depicts decoded 
node points (rectangles) and mesh boundary edge segments (straight Unes) that are the 
input to the constrained Delaunay triangulation. 

20 The mesh is finally obtained by applying constrained Delaunay 

triangulation to the set of decoded node points, where the edge segments on the mesh 
boundary are used as constraints. Delaunay triangulation proceeds by inserting an edge 
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between two node points if there exists a circle through these two points that does not 
contain any other node point in its interior and does not intersect any other node point 
The boundary edge segments present the only exceptions to this definition and may not 
be removed in the process. An example of a mesh obtained by constrained triangulation 
5 of the node points of Fig. 1 1 and is shown in Fig. 12. 
Dftcndinp of Nndfi Motion Vectors 

Each node point p„ of a 2D mesh numbered * in the sequence of meshes 
has a 2D motion vector v^ , defined from mesh k to k-^l. By decoding all motion vectors 
, ^ = 0, N- Ij one is able to reconstruct the locations of node points p^ of the mesh 
10 numbered it+l. Note that the triangular topology of the mesh remains the same 
throughout the sequence of meshes. 

As previously explained in the section entitled "Encoding of Node Motion 
Vectors", two methods for motion coding are described. Accordingly, two methods for 
motion vector decoding will be described and are identified as Method I and Method H. 

Method I is based on a predictive coding technique where only one 
motion vector is used as a predictor. A unique ordering of the node points p^ is 
available, where n denotes the ordering number. In the case that the mesh represents a 
Delaunay-type mesh, this ordering is simply defined as the order in which the node point 
20 locations are received during mesh geometry decoding. The node point for which the 
location was received first is po , the node point for which the location was received after 
that is Pp etc. In the case where the mesh is a uniform-type mesh, the node ordering is 
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defined on the basis of the mesh structure as previously described m the section entitled 
"Encoding of Node Motion Vectors: Method 1." This ordering, defined by a simple 
traversal of the uniform mesh, is illustrated in Fig. 8. 

For each node point in the mesh, a one bit flag specifies whether that node 
point has a non-zero motion vector. For each node point in the mesh with non-zero 
motion vector, a motion vector difference vector e, specifies the difference between the 
motion vector of that node and the previously decoded motion veaor. The decoding 
process is as follows: To decode the motion vectors, vectors e. are retrieved from tiie bit 
stream one by one, and used to compute the new locations of node points in the order 
specified above. If the first node point in the ordering has a zero motion vector, then: 

%=(0,0). (9) 

If the first node point in the ordering has a non-zero motion vector, then 
the first vector retrieved from the bit stteam, e^ , and specifies the motion vector of tiie 
first node point directiy: 

Vo = ^. (10) 
The new location of the first node point is obtained by: 

Po' = Pb+V (11) 
All other vectors retrieved firom tiie bit stream e^ are motion vector difference vectors 
and are added to tiie previously decoded motion vector to obtain the actual motion 
vector. If node point p„, n = 1, - 1, has a zero motion vector, then: 

v. = (0.0), (12) 

otherwise: 
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v, = v,+e„ . (13) 

The new location of node points p^, ^ = 1 - 1, is obtained by: 

P: = P„ + \- (14) 

MgthQ^ H; MQTion Vfiptor Prediction 

5 Method n is based on a predictive coding technique where two motion 

vectors are used to compute a predictor. The predictive decoding technique of Method H 
specifically employs the following principle: To decode die motion vector of a node 
point p„ that is part of a triangle t^ = (Pp P^,. Pn)» where the two motion vectors vectors 
Vi and v^ of the nodes Pj and p^ have already been decoded, the values of v, and v^ 

10 may be used to predict \ and decode the prediction error value. Starting from an initial 
triangle tj^, of which all three node motion vectors have been decoded, there must be at 
least one other, neighboring, triangle t^ that has two nodes m common with tj^. Because 
the motion vectors of the two nodes that tj^ and t^ have in common have already been 
decoded, these two motion vectors may be used to predict the motion vector of the third 

15 node in t^. The actual prediction vector w^ is computed by averaging of the two 
prediction motion vectors and the components of the prediction vector are rounded to 
half-pixel accuracy, as follows 

w =0.5x(Lv,+v^+0.5j). (15) 

The motion vector itself is computed by: 
20 v^ = w,+e„. (16) 
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where denotes the prediction error vector, the components of which are decoded from 
variable length codes. This procedure is repeated while traversing the triangles and 
nodes of the mesh, as explained below. While visiting all triangles of the mesh, the 
motion vector data of each node is decoded from the bit stream one by one. Note that no 
5 prediction is used to decode the first motion vector, 

v.=e, . (17) 

and that only the first encoded motion vector is used as a predictor to encode the second 
motion vector, 

v,=v,.4.e.,. (18) 

10 It should be noted that the prediction error vector is specified only for 

node points with a non-zero motion vector. For all other node points, the motion vector 

is simply v^ =(0,0). 

Mfithod TT: Mfn^h Trgygrsal 

A breadth-first traversal is used to visit all the triangles and nodes in the 

15 mesh numbered K and to decode the motion vectors defined from mesh klok-^l. The 
breadth-first traversal of the triangles is defined in the section on "Encoding of Node 
Motion Vectors: Method II," and may be applied similarly to the decoding phase. 

The ordering of the triangles according to their label numbers assigned 
during the breadth-first traversal implicitly defines the order in which the motion vector 

20 data of each node point is decoded, as described in the following. Initially, motion 
vector data for the top-left node of the mesh is retrieved from the bit stream. No 
prediction is used for the motion vector of this node, hence this data specifies the motion 
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vector itself. Motion vector data for the second node, which is the next clockwise node 
on the boundary with respect to the top-left node, is retrieved from the bit stream. This 
data contains the prediction error for the motion vector of this node, where the motion 
vector of the top-left node is used as a prediction. These first two nodes (that fomi the 
5 base edge of the initial triangle) are marked with the label 'done'. 

During each iteration of the breadth-first traversal as described before and 
starting from the initial triangle, the motion vectors of the two nodes that are on the base 
edge of the current triangle during that iteration are used to form a prediction for the 
motion vector of the third node of that triangle. If that third node is not yet labeled 

10 'done', motion vector data is retrieved from the bit stream and used as prediction error 
values, i.e., the decoded values are added to the prediction to obtain the actual motion 
veaor. That third node is then labeled *done\ If the third note is already labeled *done% 
then it is simply ignored and no data is retrieved from the bit stream. Due to the nature 
of the traversal process, the two nodes on the base edge of a triangle are guaranteed to be 

15 labeled 'done' when that triangle becomes the 'current triangle' , signifying that their 
motion vectors have already been decoded and may be used as predictors. 

Because every triangle in the mesh has at least one adjacent neighbor, and 
triangles are labeled when visited, every triangle is visited exactly once and the traversal 
terminates when (and only when) all triangles have been visited. Node points may be 

20 visited more than once, but their motion vectors are decoded only at the first time a node 
is visited and each node is labeled at the time of decoding, therefore, no attempt wiU be 
made to decode the corresponding motion vector again. The unique ordering of node 
points corresponds to the order of motion vector data present in the bit stream. 
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Similar to the encoding phase, the breadth-first traversal of the triangles 
and the decoding of the node motion vector data may be performed simultaneously by 
making use of a first-in-first-out (FIFO) queue of triangles. For details, see the section 
on "Encoding of Node Motion Vectors," above, and Fig. 10, which illustrate the breadth- 
first traversal process during decoding for a small triangular mesh. Fig. 10 depicts a 
traversal of triangles arbitrarily numbered t^, and the corresponding ordering of 

node points arbitrarily numbered Po p^ . The initial triangle, , is defined using the 

top-left node Pj and the next clockwise node on the boundary p^. The motion vector V3 
is decoded without using any prediction. The motion vector v^ is decoded using only V3 
as a prediction. Nodes P3 and pi are marked 'done'. The triangle queue is initialized 
with tj , and is then labeled. 

The actual breadth-first traversal of the mesh starts by removing the front 
element from the queue, iu this case . The motion vector of its third node, % , is now 
decoded using V3 and v^ as predictors and retrieving the prediction error from the bit 
stream. Node pg is marked *done\ The next step is to append any triangles adjacent to 
the current triangle that are not yet labeled to the triangle queue. The triangle on the 
right of the current triangle (in this case tg) is appended first; then the triangle on the left 
(in this case to); both are labeled. The next iteration of the traversal commences by 
removing the front element from the queue, in this case tg . The motion vector of its third 
node, V7 , is now decoded usiag V3 and v^ as predictors and retrieving the prediction error 
from the bit stream. Node p? is marked 'done' and the comiected triangle tg is appended 
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to the triangle queue and labeled. Such iterations continue until the triangle queue is 
empty and all node motion vectors have been decoded; the steps of the algorithm are also 
illustrated in Table 2, 



decode 


triangle queue state 






Ve 


{%. u 


V7 


{to. U 


Vo 


{ts. y 


Va 




V4 




V9 






{ti. u 












{} 



Table 2 



The Bit Stream Syntax and Semantics 

The following is an implementation of the invention within the current 
version of MPEG-4 Working Draft (WD )V 5.0. The Working Draft specifies only the 
decoding process. The following is an object-oriented pseudo computer code 
implementing the decoding process for both the mesh geometry and motion vectors 
according to the structure of the current specification of MPEG-4 WD V 5.0, 
Mesh Objfigt 

The Mesh Object class defines the syntax for a sequence of 2D meshes. 
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Me&hObiectO f 


No. of bits 


Mnemonic 


mesh_object_start_code 


32 


bslbf 


do{ 






MeshObjectPlaneO 






} wiiile (uextbits^yteaiignedO = 
mesfa_object_pIane_siart_code) 






next_start_codeO 






mesh_object_end_code 


32 


bslbf 


} 







Table 3 



Me.sh Object Plane 

The Mesh Object Plane class defines the syntax for coding a single 2D 



mesh, either as a new mesh or with respect to the previously coded mesh. 



MeshObjectPlaneO { 


No. of bits 


Mnemonic 


mesh_object_plane_start_code 


32 


bslbf 


new_mesh_flag 


1 


bsblf 


if (new_mesh_flag == '1') { 






MeshGeometryO 






} 






else { 






MeshModonO 






} 






} 







Table 4 



The Mesh Geometry class defines the syntax for coding the 2D geometry of a new 
mesh, which may either be a uniform mesh or a Delaunay triangulated mesh. 
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MeshGeometryO { 


No. of bits 


Mnemonic 


mesh type .code 


2 


bsibf 


if (mesh type^code = *00') { 






nr of mesh_nodes_hori 


10 


uimsbf 


nr_of_mesh_nodes_v€rti 


10 


uimsbf 


mesh rect size hori 


8 


uimsbf 


mesh rect size^verti 


8 


uimsbf 


triangle^split^code 


2 


bsibf 


} 






else if (mesh_type_code = *0r) { 






Tir_nf_mesh nodes 


16 


uimsbf 


iir_of_boundary_nodes 


10 


uimsbf 


nodeO^ 


10 


uimsbf 


nodeO_y 


10 


uimsbf 


for {n=l; ii< nr_of_mesh_nodes; 11++) { 






delta_x_len_vlc 


2-9 


vlclbf 


delta_x 


0-11 


vlclbf 


delta _y len_vlc 


2-9 


vlcibf 


deltajr 


0-11 


vlclbf 


} 






} 






} 







Table 5 



The Mesh Motion class defines the syntax for coding the 2D motion vectors of the previous 2D 
5 mesh to the current mesh, thereby coding the current 2D mesh. 
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MeshMotionO { 


No- of bits 


Mnemonic 


motion_raiige_cod€ 


2 


uimsbf 


for (n=0; n < iir_of_niesh_no<ies; b+h-) { 






node_motion_vector_£lag 


1 


bslbf 


if (no<ie_niotion_vector_flag ='!'){ 






deita_niv„3^vlc 


1-13 


vlclbf 


delta^nnrjc.res 


0-2 


uimsbf 


delta_invjr_vlc 


1-13 


vlclbf 


dclta_mvjr_res 


0-2 


uimsbf 


} 






} 






} 







Table 6 



Semantics of Syntax 

5 The mesh bit stream syntax consists of two parts: mesh geometry and 

mesh motion. The mesh geometry is only encoded when a new mesh needs to be 
initialized; it consists of the initial positions of the mesh nodes. The mesh motion is 
encoded at subsequent time instants to describe the motion of the dynamic mesh; it 
consists of a motion vector for each mesh node. 

10 The dynamic mesh syntax allows for coding different types of 2D mesh 

structures, e.g., uniform or Delaunay-constrained and different magnitude ranges for the 
node motion vectors. The node coordinate and node motion vector accuracy is 0.5 pixel, 
marker^bit - This one-bit code is set to "1". Setting this bit to "1" prevents emulation 
of start codes. 

15 mesh_object_start_code 

This is the bit string *000001BC' in hexadechnal. It marks the start of a mesh object 
This is a unique bit string of length 32 bits for the purpose of synchronization. 
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mesh_object_end_code 

This is the bit string in hexadecimal. It marks the end of a mesh object. This is a unique 
code of length 32 bits for the purpose of synchronization. 
mesh_object_plane_start_code 
5 This is the bit string *000001BD* in hexadecimal. It marks the start of a mesh object 
plane. This is a unique code of length 32 bits for the purpose of synchronization. 
new_mesh_flag 

This is a one bit code specifying whether a new mesh is to be initialized or that the 
current mesh is encoded with respect to the previous mesh. 



new_mesh_flag 


mesh coding type 


1 
0 


I (coded by geometry) 

P (coded by motion with respect to previous mesh 



10 Table 7 

mesh_type_code 

This is a one bit code specifying the type of initial mesh geometry being encoded. 



mesh_type_code 


mesh geometry type 


00 


uniforai 


01 


Deiaunay 


10 


not used (reserved) 


11 


not used (reserved) 


Tables 



15 

nr_of_mesh_nodesJhori 

This is a 10 bit code specifying the number of nodes in one row of a uniform mesh. 
nr_of_mesh_nodes_ver1i 

This is a 10 bit code specifying the number of nodes in one column of a uniform mesh. 
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mesh_rect_size_hori 

This is a 8 bit code specifying the width of a rectangle of a uniform mesh (containing 

two triangles) in half pixel units, 

mesh_rect_size_ver1i 

5 This is a 8 bit code specifying the height of a rectangle of a uniform mesh (containing 
two triangles) in half pixel units. 
triangie_spIit_code 

This is a 2 bit code specifying how rectangles of a uniform mesh are split to form 
triangles. 



triangle_split_code 


type 


00 
01 
10 
11 


split all rectangles top-left to bottom-right 
split ail rectangles bottom-left to top-rigiit 

split alternately top-left to bottom-right and bottom-left to top-right 
split alternately bottom-left to top-right and top-left to bottom-right 



10 Table 9 

nr_of_mesh_nodes 

This is a 16 bit code defining the total number of nodes (vertices) of a (non-uniform) 
Delaunay mesh. These nodes include both interior nodes as- well as boimdary nodes. 
nr_of_boundary_nodes 



15 This is a 10 bit code defining the number of nodes (vertices) on the boundary of a (non- 
uniform) Delaunay mesh. 
nodeO_x 

This is a 10 bit code specifying the x-coordinate of the first boundary node (vertex) of a 
mesh in half-pixel units with respect to a local coordinate system. 
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nodeO _y 

This is a 10 bit code specifying the y-coordinate of the first boundary node (vertex) of a 

mesh in half-pixel units with respect to a local coordinate system. 

delta_x_Ien_vlc 

5 This is a variable length code specifying the length of the delta_x code that follows. The 
delta^x Jen.vlc and delta^x codes together specify the difference between the x- 
coordinates of a node (vertex) and the previously encoded node (vertex). The definition 
of the deltajxjen^vlc and delta^x codes are given in the MPEG-4 Video Verification 
Model 7.0 (VM 7.0) (N1642, section 4.7.1.1) (April 1997), hereinafter MPEG-4 WM 

10 7.0, where they are used for sprite trajectory coding. 
delta_x 

This is a fixed length code defining the value of the difference between the x-coordinates 
of a node (vertex) and the previously encoded node (vertex). The definition of the 
delta_xjen_vlc and de!ta_x codes are given in the MPEG-4 WM 7.0, section 4.7.1.1, 
15 where they are used for sprite trajectory coding. 
delta_y_Ien_vlc 

This is a variable length code specifying the length of the delta_y code that follows. The 
delta j^Jen_vlc and delta_y codes together specify the difference between the y- 
coordinates of a node (vertex) and the previously encoded node (vertex). The definition 
20 of the d€lta_yjen_vlc and delta^y codes are given in the MPEG-4 WM 7.0, section 
4.7.1. 1, where they are used for sprite trajectory coding. 



37 



wo 98/27515 



PCT/JP97/04607 



delta_y 

This is a fixed length code defining the value of the difference between the y-coordinates 
of a node (vertex) and the previously encoded node (vertex). The definition of the 
deltajrjen.vlc and delta^y codes are given m the MPEG-4 WM 7.0, section 4.7.1.1, 
5 where they are used for sprite trajectory coding. 
motion_range_code 

This is a 2 bit code specifying the dynamic range of motion vectors. 



motion_raiige_code 


motion vector range in 


half sample units 


1 


[-32. 31] 


2 


[-64, 63] 


3 


r-128. 1271 


Table 10 



node_motion_vector_flag 



10 This is a 1 bit code specifying whether a motion vector follows. If a node has a zero 
motion vector, then the motion vector is not encoded; instead, a 
node_motion_vector_flag with value "0" is sent If this flag is set to it means the 
node has a non-zero motion vector. 
delta_mv_x_vlc 

15 This is a variable length code defining (together with delta_mv_x_res) the value of the 
difference in the x-component of the motion vector of a node compared to the x- 
component of a predicting motion vector. The section on the decoding process, describes 
how the predicting motion vector is found. The definition of the delta_mv_x_vlc and 
delta^mv^x^res codes are given in the MPEG-4 WM 7.0, section 6.1.8 and 6.1.9, 

20 Table 30, where they are used for motion vector difference coding. 
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delta_mv_x.res 

This is a fixed length code defining the residual in the difference of motion vectors as 
encoded by delta_mv_x_vlc. The definition of the delta_mv_x_vlc and 
delta_mv_x„res codes are given in the MPEG-4 WM 7,0, section 6,1.8 and 6.1.9, 
5 where they are used for motion vector difference coding. 
delta„inv_y_vlc 

This is a variable length code defining (together with delta_mv_y_res) the value of the 
difference in the y-component of the motion vector of a node compared to the y- 
component of a predicting motion vector. The section on the decoding process describes 
10 how the predicting motion vector is found. The definition of the delta_mv_y_vlc and 
delta_mvjr_res codes are given in the MPEG-4 WM 7.0 section 6.1.8 and 6.1.9, Table 
30, where they are used for motion vector difference coding. 
delta_inv_y_res 

This is a fixed length code defining the residual in die difference of motion vectors as 
15 encoded by delta_mv_y_vlc. The definition of - the delta_mv_y_vlc and 
delta_mv_y_res codes are given in the MPEG-4 WM 7.0 section 6.1.8 and 6.1.9, where 

they are used for motion vector difference coding. 

Tndn.strial Applications 
The 2D dynamic mesh representation supports the foUowuig applications: 
20 Video Objec t Manipulation 

In 3D computer graphics, the animation parameters of a 3D mesh model 
are often synthetically specified. The animation parameters of a 2D mesh model may 
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likewise be specified syntheticaUy, but they may also be derived from a natural video 
sequence by motion estimation. Thus, mesh modeling allows us to interactively combine 
natural and synthetic objects within a unified framewo±. Possible applications include 
object transfiguration (replacing one object by another object in moving video), 

5 augmented reaUty (augmenting/overlaying an object in moving video with graphics or 
text) and spatio/temporal image interpolation (zooming or frame-rate conversion) 
Vidfln rnmpression 

Motion estimation and compensation have proven to be important tools 
for video coding systems to obtain high compression ratios. The properties of the mesh- 

10 based motion model make it a suitable alternative to block-based motion modeling for 
use as a motion compensation tool in video coding in which case die motion vectors of 
mesh nodes are transmitted instead of block-motion vectors. Mesh-based video 
compression may for instance be performed by transmitting texture maps only at selected 
key frames and by animating these texture maps (without sending any prediction error 

15 unage) for the intermediate frames using 2D mesh information. Only the image data of 
the first frame of a sequence is encoded, as well as the mesh motion data corresponding 
to all frames of the sequence. In case the initial mesh is adapted to image content, 
information about the initial mesh geometry has to be encoded and transmitted as well. 
In the case of video coding, the initial mesh geometry is sometimes restricted to limit the 

20 overhead involved (e.g., uniform meshes); however, more general mesh geometry 
compression schemes may be applied in case of multimedia coding, where several 
objects (video, audio, text, 2D / 3D meshes etc.) are encoded independently. 
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Indexing and T^rowsing for Video Libraries 

Content-based random access to individual video objects becomes a 
desirable feature in several distributed multimedia applications, such as object-baseci 
browsing/editing/manipulation of video databases. Useful motion and shape features 
may be derived from the 2D content-based triangular mesh representation of video which 
may be employed in such applications. 

Thus, a system and method for a 2-D mesh geometry and motion vector 
compression have been disclosed. Although a preferred embodiment of the method of 
the invention, and several variations thereof, have been disclosed, it should be 
appreciated that further variations and modifications may be made thereto without 
departing from the scope of the invention as defined in the appended claims. 
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CLAIMS 

1. A method of encoding and decoding a dynamic mesh, comprising: 
encoding and decoding a mesh geometry of a set of node points; and 
encoding and decoding a mesh node motion vector for each node point 

2. The method of claim 1 wherein said encoding and decoding of a mesh 
geometry includes compressing a mesh topology of a Delamiay type. 

3. The method of claim 1 wherein said encoding and decoding of a mesh 
geometry includes compressing a mesh topology of a uniform type. 

4. The method of claim 1 wherein said encoding and decoding of a mesh 
node motion vector includes: 

compressing the motion vectors for each node point using a linear 
traversal of the node points and 

predicting, using first order prediction, subsequent motion vectors for 

each node point 

5. The method of claim 4 wherein said predicting includes encoding and 
decoding prediction errors using variable length codes. 

6. The method of claim 4 which includes traversing and ordering the mesh 
node points for a Delaunay type mesh. 
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7. The method of claim 4 which includes traversing and ordering the mesh 

node points for a uniform type mesh. 



8. The method of claim 1 wherein said encoding and decoding of a mesh 
node motion vector includes: 

compressing the motion vectors for each node point using a breadth-first 
traversal of the node points; and 

predicting, using second order prediction, subsequent motion vectors for 
each node point 

9. The method of claim 8 wherein said predicting includes encoding and 
decoding prediction errors using variable length codes. 

10. A method of encoding and decoding a dynamic mesh having a set of 
polygonal patches therein, comprising: 

encoding and decoding a mesh geometry of a set of vertices; and 
encoding and decoding a mesh node motion vector for each vertex. 

11. The method of claim 10 wherein said encoding and decoding of a mesh 
geometry includes compressing a mesh topology of a Delaunay t3rpe. 



12. The method of claim 10 wherein said encoding and decoding of a mesh 

geometry includes compressing a mesh topology of a uniform type. 
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13. The method of claim 10 wherein the polygons are triangles and said 
encoding and decoding of a mesh node motion vector includes: 

compressing the motion vectors for each vertex using a linear traversal of 
the vertices; and 

predicting, using first order prediction, subsequent motion vectors for 

each vertex. 

14. The method of claim 13 wherein said predicting includes encoding and 
decoding prediction errors using variable length codes, 

15. The method of claim 13 which includes traversing and ordering the mesh 
vertices for a Delaunay type mesh. 

16. The method of claim 13 which includes traversing and ordering the mesh 
vertices for a uniform type mesh. 

17. The method of claim 10 wherein said encoding and decoding of a mesh 
node motion vector includes: 

defining triangles throughout the mesh; 

compressing the motion vectors for each vertex using a breadth-iSrst 
traversal of the triagles and the vertices of the triangles; and 

predicting, using second order prediction, subsequent motion vectors for 

each vertex. 
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18. The method of claim 17 wherein said predicting includes encoding and 

decoding prediction errors using variable length codes. 



19, A system for encoding and decoding an image represented by a dynamic 

mesh, wherein the dynamic mesh includes a predefined geometry, having a set of node 
points and motion vectors for the node points; comprising: 

an encoder subsystem for encoding the mesh geometry of the set of node 
points and encoding a mesh node motion vector for each node point including: 

a texture encoder for encoding the texture of an image and for 
producing an image bitstream therefrom; 

a mesh encoder for encoding a mesh sequence; said mesh encoder 
including a geometry encoder for encoding the geometry of the mesh, and a motion 
vector encoder for encoding the motion vectors of the mesh; and for generating a mesh 
coded bitstream therefrom; 

a decoder subsystem for decoding the mesh geometry of the set of 
node points and decoding a mesh node motion vector for each node point including: 

a texture decoder for decoding the texture of an image from an 

image bitstream; 

a mesh decoder for decoding said mesh encoded bitstream, 
including a geometry decoder for decoding the geometry of the mesh, and a motion 
vector decoder for decoding the motion vectors of the mesh; and 

an animation application for combining the decoded texture data and the 
decoded mesh, and for generating an animated image therefrom. 
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